The image of live actors interacting with animated or pre-recorded images and sound is commonplace. In film or television, the coordination between the various images and sounds is largely achieved in the editing room, or as the result of repeated rehearsal until the timing of the combined performance is exact. In such cases, the combined image is recorded and, therefore, each viewing is identical. With pre-recorded images, unfortunately, there is no mechanism to adjust the flow of the presentation to allow for variations in the performance or the audience response. All aspects of the performance are predetermined and therefore the presentation is passive to the audience's reaction.
The use of pre-recorded audio and visual material in live performance is also commonplace in theatre, music, and business presentations. Again, however, there are significant limitations with the ability to manipulate the pre-recorded material and electronic media in coordination with the live performance. Visuals effects, sounds and other media can be linked together, but such synchronization capabilities are currently limited and typically include significant manual intervention. For example, a technician may, in response to some aspect of a live performance, manually trigger video, sound and lighting changes together. Once triggered, the various effects typically run for a fixed time. It is therefore necessary for the performers to have rehearsed their timing exactly and for the related media and control signals to have been pre-recorded and matched together exactly. For instance, once an actor has embarked on a dialogue in such circumstances, there is no mechanism to adjust the pace and/or timing of other media employed concurrently in response to variations in the actor's performance or the audience's response. Accordingly, it is believed that the use of pre-recorded media in live performance has been restricted to highly rehearsed and short sequences.
U.S. Pat. No. 5,790,124 (the '124 patent) discloses a system for allowing a live performer to interact with a background display screen that is driven by real-time generated, textured mapped graphics. By tracking the position of the performer, the system of the '124 patent can control and change the production of visual images as the performer moves on the stage to give the illusion to an audience that the performer is moving about a virtual scene. However, the system of the '124 patent is still sensitive to unexpected interruptions, disruptions, and other events that affect the timing of the performance.